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Larry H. Ludlow 

Boston College, School of Education 
Paper presented at AERA: Montreal, April 20, 1999 

STUDENT DRAWINGS AS COURSE EVALUATIONS: WHAT THEY SEE 

IN STATISTICS 




Imagine you have just presented a lecture on maximum likelihood estimation and 
the operation of the Newton-Raphson iteration technique to your doctoral students. You 
have presented this topic a half dozen times in your career. Tonight, however, you feel you 
have actually “taught” the students how the technique works and why knowing about it is 
important. You even believe you have made the topic interesting, if not exactly exciting. 
Bottom-line, you have answered the ultimate questions, “So what?” and “Who cares?” 

As you look at the students, awaiting their applause, a thought occurs, “What are 
they thinking?” You ponder this question as they file out of the room. There is no 
applause, no wave, no cheers, not even a “Nice job. Doc!” You wonder how they describe 
this class to their friends. What visual images do they construct for their audiences? 

Over the next couple of days you ask a few students what they thought about the 
lecture. Did they understand it reasonably well? Was it clear? Did it make sense? Was it at 
their level? Where were the tough parts? Where did they begin to lose it? Their responses 
are non-descript— it was fine, it was interesting to see how the parts fit together, it made 
sense at the time, it was challenging but OK. Their responses, while somewhat supportive 
of your effortSi don’t leave you satisfied. So you decide to try something unusual in the 
next class. At the start of the next class session you ask them to complete the following 
task: 

(1) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and 
anything else that represents for you that “typical” classroom experience. Ideally, 

3 



AERA 99: See Statistics: 4/15/99: p 1 



someone else could look at your drawing and could then form a reasonable 
impression of your experience. 

(2) On the back of your drawing write a full description of the scene you have 
drawn. Be as explicit, open, and comprehensive as you can. 

The class ripples with giggles. Students look at one another. Puzzled expressions are 
exchanged. Whispers and groans are heard. Some of them look at you as if you have gone 
really weird on them this time. Eventually they begin to draw. 

Theoretical Framework 

Since 1983 I have taught graduate level statistics and psychometrics courses. 
Consistent with recent calls for reform of college level statistics, particularly with respect to 
cooperative learning (Garfield, 1993; Giraud, 1997), I have tried to integrate small-group, 
interactive learning activities into the formal lecture periods. In addition. I’ve tried to 
incorporate the ideas of context-relevant material (Sowey, 1995), tried to devise alternative 
forms of testing and grading (Garfield, 1994), and tried to team teach the statistical 
software component of courses (Rumsey, 1998). Since the material changes somewhat 
from one year to the next, including the text, as do the exams, assignments, and teaching 
assistant it is problematic when I think about how to gauge the extent to which such 
pedagogical changes have been effective. Although I pass out the standard end-of-course 
, evaluation forms required at my university, it is apparent that the evaluations give me little 
information about student experiences in my classes. When I learned in 1995 that 
elementary and middle school students were drawing interesting pictures of their 
classrooms that were useful to teachers (Weber & Mitchell, 1995; Haney, et al, 1998; 
Gulek, in progress), I decided to try the drawing technique in my classes as a means of 
obtaining additional course evaluation feedback. To my amazement and delight, the 
drawings are rich beyond any expectation I held for them. 
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Figures 1 and 2 were drawn by two students at the start of one of the final sessions 
of an Intermediate Statistics II class. From my perspective, Figure 1 reflects the kind of 
positive classroom experience we wish we had on students every night we stand before 
them. The statements bursting from my head make sense. I come across as a person, even 
a personality- “Yoda.” There is a discernible message on the chalk board. The student 
doesn’t even draw any stereotypical equations, numbers, curves or tables. Most dramatic 
of all, however, is the student’s role. “Luke” is experiencing the “ah-ha” rush of insight 
that I have never seen represented as effectively as it is here. The relation implied between 
“Yoda” and “Luke” could probably comprise an article in its own right. 

This scene is even more impressive in that it was drawn at the end of the semester 
yet this particular lecture took place about two months earlier. The distinction I was trying 
to make between the mechanics of statistical analysis involving computation versus data 
analysis involving interpretation and research design seems to have been grasped by this 
student. 

Figure 2, however, is disturbing, chilling, even embarrassing. The board work and 
“statement bubble” leave no doubt that the material, maybe including me, is “BS” to this 
student. The belt buckle is even bothersome. I don’t have one like it and its pentagram 
design suggests, to me, that this is the “course from hell.” These images are not ones that 
we would like students to take away from our classes. 

^Data Sources 

I now have drawing data from seven different graduate courses that I teach— 
Interpreting & Evaluating Research (master’s students). Statistics I (master’s & PhD), 
Statistics II (master’s & PhD), Multivariate I (PhD), Multivariate II (PhD), Psychometrics 
(PhD), and Seminar in Advanced Statistical Analysis (PhD). The courses cover a four year 
time span and most of them have been taught (and drawn) more than once. Some courses 
follow a traditional lecture format (Mult I, Mult II, Psychometrics), the others include a 
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cooperative learning component. The students are all enrolled in a department of education. 
They are required to take the first three courses listed above. Students in the four higher 
level courses specialize in measurement, evaluation, and statistical analysis. 

The “drawing evaluation” is performed immediately following their completion of 
the standard evaluation form. They are told that the drawing evaluation is part of a long- 
term research project and I encourage them to take the exercise seriously. They are also 
asked to write an arbitrary four-digit code of their choice on their scannable evaluation 
forms and their drawing. They place the standard evaluations into a folder that another 
student takes to the Dean’s office. They place the drawings onto a pile that they are 
encouraged to look at and discuss after everyone has completed the task. 

Results 

The drawings offer numerous ways in which the extent to which changes in 
teaching practice have been effective. For example, I have drawings from Stat I and Stat II 
in one year prior to incorporating cooperative activities and drawings from the same two 
courses in the next year after small-group activities were introduced. One noticeable feature 
in the first drawings are the representations of the class environment in terms of traditional 
rows of seats while in the following year there are numerous drawings of groups of seats 
with arrows representing lines of communication going back and forth between the seats. 

The effect of class size is seen in the Interpreting and Evaluation Research course. 

^The first three sets of drawings for this course dearly show the small-group, highly 
interactive asp^t of the course. The next sets of drawings in the sequence show the effect 
of a doubling of class size. A noticeable proportion of the drawings revert back to a 
traditional representation of rows of seats (even u uh the same small-group component). 

Other aspects of teaching practice are possible to investigate. How is the instructor 
represented— do I face the students or the board ? What board work do the students 
represent— is it a mess of equations or plots and graphs? What resources are represented— is 
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it predominately a board or are there computers, texts, projectors, and manuals? How is 
change in learning represented— a drawing with three parts labeled “wkl” with three 
question marks over a stick figure of a student, then “wk3” with two question marks, 
finally “wk5” with a question mark scratched through and a little light bulb drawn? How do 
the drawings differ as a function of statistical sophistication— do they become more 
coherent, focused, technical, and interesting as the students progress through the statistics 
sequence? Are there particular topics that tend to be drawn for specific classes— do z scores 
stand out in Statistics I drawings while matrix operations stand out in Multivariate I? 

The drawings also provide an opportunity to understand how students perceive 
their peers. The perceptions students have of their peers is beautifully illustrated in a 
Multivariate I drawing where groups of counseling psychology, human development, and 
measurement, evaluation and statistical analysis students are seated together in different 
regions of the class. It is not unusual that students sit with their program peers but it is 
noteworthy when the drawings also contain images of dead fish over one group, question 
marks over another, and three dimensional box-like diagrams with vector projections over 
the other. Because some of these students were enrolled in multiple courses with me it is 
even possible to identify individual students who were drawn by their peers as particularly 
noteworthy participants in class discussions. 

Educational Importance 

Included with the drawing exercises were instructions to describe the scenes and to 
offer their insights into what their drawings have to offer in the way of course evaluation 
information that the scannable forms were not able to convey. The following written 
statement by a student illustrates the point of this research: “This type of evaluation may get 
to the heart of the class-what the atmosphere is like-that’s what can make or break a class!” 
In addition, the personal codes they provided on their standard forms and drawings provide 
a means to investigate the relation between quantitative ratings and qualitative drawings. 
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Finally, it is possible to investigate instructor ratings in the same course before- and-af ter 
implementation of cooperative learning activities. 

All of us who teach statistics, particularly to non-specialists in departments of 
education, are well aware of the negative attitude and belief systems that many students 
carry as baggage to required statistics courses (Gal & Ginsburg, 1994; Yilmaz, 1996). 
Those attitudes and beliefs are not necessarily well communicated on standard course 
evaluation forms. Even more of a problem from our standpoint as instmctors is the inability 
of students to effectively articulate changes in negative attitudes and beliefs. Scenes with 
“ah-ha’s”, dead fish, light bulbs, and tear-drops effectively communicate what students 
have felt in my classes. Those impressions make me think about what I now do in class in 
ways that never occurred to me before. 

Although there are issues of reliability and validity in the interpretation of these 
drawings, these drawings do encourage self-reflection and inquiry. While I recognize the 
dangers of self-promotion, over-interpretation, exaggeration and other forms of subjective 
bias, I also believe that the gestalt portrayed in these drawings is so powerful that they offer 
a unique opportunity to combine qualitative data with quantitative course evaluation data in 
order to yield a richer understanding of the psychological dynamics underlying student 
experiences in class. 

Format: 

Roundtable consisting of sets of drawings to illustrate various aspects of student 
experiences and their implication for teaching practice. 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 

( 1 ) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is confidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that I am conducting on alternative modes of faculty evaluation 
assessment techniques. 
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DRAWING INSTRUCTIONS W G rl ^ 



Think of a “typical” classroom experience this semester with me. Now draw as best 
as you can, that classroom experience. Include me, yourself, and everything else that 
represents for you that “typicd” classroom experience. 

On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

In addition, what “course evaluation” information do these drawings offer that the 
traditional scaimable forms do not contain? 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 

(1) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom expaience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 



Please try to accept my assurance to you that this information is corrfidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that I am conducting on alternative modes of faculty evaluation 
assessment techniques. 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 

(1) Think of a “typical” classroom teaching experience this semester with me. Now 
^aw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is confidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that I am conducting on alternative modes of faculty evaluation 
assessment techniques. 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 

(1) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is confidential-I will not try tc 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that l am conducting on alternative nwdes of faculty evaluation 
assessment techniques. 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 

( 1 ) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is oxifidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that I am conducting on alternative modes of faculty evaluation 
assessment techniques. 
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Think of a “typical” classroom experience this semester with me. Now draw as best 
as you can, that classroom experience. Include me, yourself, and everything else that 
represents for you that “typical” classroom experience. 

On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

In addition, what “course evaluation” information do these drawings offer that the 
traditional scannable forms do not contain? 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 



( 1 ) Think of a “typical” classroom teaching experience'this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is confidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that I am conducting on alternative modes of faculty evaluation 
assessment techniques. 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 



(1) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, ycHirself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 

Please try to accept my assurance to you that this information is confidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
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DRAWING INSTRUCTIONS FOR COURSE EVALUATION 



(1) Think of a “typical” classroom teaching experience this semester with me. Now 
draw as best as you can, that classroom experience. Include me, yourself, and anything 
else that represents for you that “typical” classroom experience. Ideally, someone else 
could look at your drawing and could then form a reasonable impression of your 
experience. 

(2) On the back of your drawing write a full description of the scene you have drawn. 
Be as explicit, open, and comprehensive as you can. 

(3) Finally, what “course evaluation” information does your drawing provide that your 
responses to the traditional scannable form do not contain? 



Please try to accept my assurance to you that this information is Confidential-I will not try to 
somehow figure out who passed in which one of these sheets. This information is part of a 
long-term research project that l am conducting on alternative modes of faculty evaluation 
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